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DETAILED ACTION 



1 . This action is responsive to communications: Amendment A filed 01/1 6/2004 to the 
original application filed 01/28/2000. 

2. Claims 1-21 are currently pending in this application. Claims 1, 4, 9 and 13 are 
independent claims. 

3. If a copy of a provisional application listed on the bottom portion of the accompanying 
Notice of References Cited (PTO-892) form is not included with this Office action and the PTO- 
892 has been annotated to indicate that the copy was not readily available, it is because the copy 
could not be readily obtained when the Office action was mailed. Should applicant desire a copy 
of such a provisional application, applicant should promptly request the copy from the Office of 
Public Records (OPR) in accordant with 37 CFP. I.14(a)(l)(iv), paying the required fee under 
37 CFR 1.19(b)(1). If a copy is ordered from OPR, the shortened statutory period for reply to 
this Office action will not be reset under MPEP § 710.06 unless applicant can demonstrate a 
substantial delay by the Office in fulfilling the order for the copy of the provisional application. 
Where the applicant has been notified on the PTO-892 that a copy of the provisional application 
is not readily available, the provision of MPEP § 707.05(a) that a copy of the cited reference will 
be automatically furnished without charge does not apply. 
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Rejections - 35 USC § 102 



4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b),by another filed in the 
United States before the invention by the applicant for patent or (2)a patent granted on an application for patent by 
another filed in the United States before the invention by the applicant for patent, except that an international 
application filed under the treaty defined in section 351(a)shall have the effects for the purposes of this subsection of an 
application filed in the United States only if the international application designated the United States and was 
published under Article 21(2)of such treaty in the English language; or " (Emphasis added.) 

Claims 1-2, 4-5, 7-9 and 13-18 remain rejected under 35 U.S.C. 102(e) as being 
anticipated by Smith et al. (U.S. 6,223,183). 

As to independent claim 1, Smith teaches a method of describing a multiple level digest 
segment information scheme for multimedia contents in order to provide multiple levels of digest 
streams for each multimedia content with small amount of additional siorage (uniformly 
describing space and frequency views... including regions, tilings and hierarchical 
decompositions of image, video, audio content, and time series data in space, time, frequency 
and resolution. The space and frequency view description scheme provides a way to specify 
regions in space, time, frequency and resolution in term of space and frequency views; Abstract) 
in accordance with an embodiment comprising the steps of: 

- describing the level information of digest segments by multiple levels in the 
content-based data area of the multimedia stream (describe any arbitrary multi-resolution 
decomposition of any number of levels of the image or video data; col. 6, lines 41-55); 
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- describing the digest level information and the time range information of each digest 
segment in a digest segment information structure (the space and frequency view description 
scheme defines several object classes for specie regions in multi-dimensional rectangular region 
in space and frequency ...to describe locations, durations, sizes and regions in space and 
frequency, col. 7, lines 9-30/ The view requests can be represented in the form ofSFViews to 
provide a uniform, standard interface for specifying the space, time, frequency and resolution 
parameters of the views; col. 11, lines 15-23); and 

- describing digest segment information scheme with a set of digest segment information 
structures (the space and frequency view description scheme can provide an abstraction layer 
between image, video and audio description schemes; col. 10, lines 30-50/it can be important to 
describe the color of different regions of the image... The space and frequency view description 
scheme can be used to specify each of the regions in terms of SFViews, and by associating a 
color measurement function with the space and frequency view description scheme; col. 12, lines 
10 13 and Fig. 15). 

As to dependent claim 2, Smith teaches the time range information is the start point and 
end point of or the start point and duration of the digest segment (col A, lines 13-65). 

As to independent claim 4, the rejection of independent claim 1 above is incorporated 
herein in full. However, claim 4 further recites "a digest stream information scheme with a set of 
digest level headers." 

Smith teaches a digest stream information scheme with a set of digest level headers 
(col 8, lines 32-45 & Fig. 10). 
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As to dependent claim 5, it includes the same limitations as in claim 2, and is similarly 
rejected under the same rationale. 

As to dependent claim 7, Smith teaches the digest level headers can be arranged with the 
order of importance (level) in order to construct a digest stream from multi level digest segment 
information scheme fast (Fig. 10). 

As to dependent claim 8, Smith teaches the digest level segment information structures 
can be arranged with the order of their time range information in order to construct a digest 
stream from multi level digest segment information scheme fast (col. 5, lines 13-65). 

As to independent claim 9, Smith teaches a method of generating multiple levels of 
digest streams for multimedia contents {uniformly describing space and frequency views... 
including regions, tilings and hierarchical decompositions of image, video, audio content, and 
time series data in space, time, frequency and resolution. The space and frequency view 
description scheme provides a way to specify regions in space, time, frequency and resuiuiion in 
term of space and frequency views; Abstract) in accordance with the present invention 
comprising the steps of: 

- detecting the digest level and time range information of the digest segment information 
structures from the multiple level digest stream information scheme contained in the 
content-based data area of the multimedia stream (the space and frequency view description 
scheme defines several object classes for specie regions in multidimensional rectangular region 
in space and frequency ...to describe locations, durations, sizes and regions in space and 
frequency; col. 7, lines 9-30/ The view requests can be represented in the form ofSFViews to 
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provide a uniform, standard interface for specifying the space, time, frequency and resolution 
parameters of the views; col 11, lines 15-23); 

- when a condition is queried by the user, generating a multiple level digest stream by 
arranging the digest segments with a priority of more than a certain level corresponding to the 
condition in a time sequence {the client can send the requests for different multi-resolution sub- 
region views to the server. The server can respond by retrieving the views from storage and 
transmitting the results to the client. The view requests can be represented in the form of 
SFViews to provide a uniform, standard interface for specifying the space, time, frequency and 
resolution parameters of the views. The data at the server can also be represented using 
SFPartitionings or SFHierarchical decompositions to provide a standard way for accessing the 
views; col 10, line 65-col.ll, line 23). 

As to independent claim 13, the rejection of independent claim 9 above is incorporated 
herein in full. However, claim 13 recites: 

- ci user input uiiiij and 

- a decoder for decoding digest segments of the above digest level from the multimedia 
stream signal. 

Smith teaches: 

- a user input unit (a client application can send requests for views of images; abstract); 

- a decoder for decoding digest segments of the above digest level from the multimedia 
stream signal (The view requests can be represented in the form of SFViews to provide a 
uniform, standard interface for specifying the space, time, frequency and resolution parameters 
of the views. The data at the server can also be represented using SFPartitionings or 
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SFHierarchical decompositions to provide a standard way for accessing the views; colli, lines 
14-23), 

As to dependent claim 14, Smith teaches the condition is the running time of a digest 
stream (col 5, lines 13-53). 

As to dependent claim 15, Smith teaches the condition is one of a time constraint of 
digest stream, a level information of digest stream, an occurrence of an event, an appearance of a 
person, a background, an object, situation information about of event, person, object, background 
(col.5, lines 13-53 & Figs. 12-14). 

As to dependent claim 16, Smith teaches the condition is combination of a time 
constraint of digest stream, a level information of digest stream, occurrence of events, 
appearance of persons, backgrounds, objects, situation information about of events, persons, 
objects, backgrounds (col5, lines 13-53 & Figs. 12-1 4). 

As to dependent claim 17, Smith teaches a digest level information and a time range 
information, and/or the information of the sum ot the running time uf cawi digest segment are 
contained in the digest level (col 3, lines 14-20 & col 5, lines 13-53). 

As to dependent claim 18, it includes the same limitations as in claim 2, and is similarly 
rejected under the same rationale. 

Allowable Subject Matter 

4. Claims 3, 6, 10-12 and 19-21 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the limitations of 
the base claim and any intervening claims. 
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Response to Arguments 

5. Applicant's arguments filed 01/06/2004 have been considered but they are not 
persuasive. 

Applicant argues that because the U.S. Patent and Trademark Office has not provided the 
Smith provisional application, the outstanding rejection based upon the Smith patent should be 
withdrawn. (Remarks, page 10) 

In response, a copy of the Smith provisional application is provided, and Smith reference 
is remained for rejection. 

Conclusion 

6. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

7. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is 
reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
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however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Maikhanh Nguyen whose telephone number is (703) 306-0092. 
The examiner can normally be reached on Monday - Friday from 9:00am - 5:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on (703) 305-9792. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 

svstem nnntap.t thp FWtrnrnr RiiqWcc Cmit>r fFRf^ *t 7.01 07 rtnll.-fVml 



Maikhanh Nguyen 
June 28, 2004 
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Method for Describing Views in Space, Time, Frequency, 

and Resolution 



John R. Smith and Chung-Sheng Li 
Related Applications 

The following commonly assigned, co-pending patent applications are hereby incorporated by 
reference in their entirety: 

S/N 09/079,986 entitled "Interactive Retrieval and Caching of Multi-dimensional Data," 

by Castelli et al, IBM Docket No. Y0998191; and 
S/N 09/079,662 entitled"Interactive Representation and Retrieval of Multi-dimensional 
Data Using View Elements," by Castelli et al., IBM Docket No. Y09981 1 1 



Field of the Invention 

The present invention relates to a method for describing locations, sizes, regions, tilings 
and hierarchical decompositions of image, video and audio content in space, time, frequency and 
resolution; for providing an abstraction layer between image, video and audio description 
schemes and multimedia applications and the stored, compressed data, which allows the data to 
be referenced and accessed in terms of space and frequency views; for providing a view 
indexing system using a space and frequency graph (SFGraph); for communicating view requests 
and descriptions between client and server in retrieval applications; for describing 
multi-dimensional shapes in terms of sets of views; for adding attributes to objects, annotations, 
meta-data and descriptors that specifies information about locations, sizes, and regions in space, 
time, frequency and resolution; and for specifying the derivation of descriptors that utilize 
statistical measures computed from regions in space, time, frequency and resolution. 
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Background 

MPEG-7 is standardizing the interface for multimedia content search engines and 
filtering agents. The objective is to encourage the creation of standard meta-data to be published 
with the multimedia content. This will improve the ability by which the content can be indexed, 
searched, browsed, filed and filtered in a large number of multimedia storage and retrieval 
applications. 

Towards this end, multimedia search, retrieval and filtering applications need a standard 
reference system for dealing with regions, locations and sizes along the dimensions of space, 
time, resolution, and frequency. In many cases, they need to deal with these dimensions 
explicitly, such as when referring to spatial regions of images, temporal units of video, 
low-resolution versions of images, or spectral bands of audio. The applications also need to be 
able to refer to regions and locations in space, time, resolution and frequency without concern for 
the underlying storage and compression formats for the image, video and audio data. The 
applications need this higher-level interface to insulate them from the specific details of the data 
bit-streams and view extraction methods. In order to enable this abstraction, a standardized 
description scheme is needed for specifying location, size and division along the dimensions of 
space, time, frequency and resolution as they pertain to the image, video and audio data, and for 
relating the views of the data with the data storage formats. 

Independent of the storage and compression formats, digital multimedia content such as 

images, video and audio has an inherent lattice structure. For example, image data consists of 

pixels that are samples on a 2-D spatial grid. Similarly, audio and video data consists of samples 

in time, or in space and time, respectively. Spatial and temporal views, such as spatial quadrants 

of an image or temporal segments of video correspond to the data in parts of the lattice. 
YO999-043 2 



Frequency views, such as low-resolution images, wavelet subbands, or temporal-frequency 
channels correspond to segments of the data after transformed into the frequency domain. This 
allows us to develop a general notion of views of image, video and audio content - a view is a 
region in multi-dimensional space (includes time) and multi-dimensional frequency (includes 
resolution) that has location and size. Since multimedia applications need to deal with views all 
the time, a standard description scheme (DS) is needed to describe views. 

Summary of The Invention 

In accordance with the aforementioned needs, the present invention is directed towards a 
method for describing locations, sizes, regions, tilings and hierarchical decompositions of image, 
video and audio content in space, time, frequency and resolution; for providing an abstraction 
layer between image, video and audio description schemes and multimedia applications and the 
stored, compressed data, which allows the data to be referenced and accessed in terms of space 
and frequency views; tor providing a view indexing system using a space and frequency graph 
(SFGraph); for communicating view requests and descriptions between client and server in 
retrieval applications; for describing multi-dimensional shapes in terms of sets of views; for 
adding attributes to objects, annotations, meta-data and descriptors that specifies information 
about locations, sizes, and regions in space, time, frequency and resolution; and for specifying 
the derivation of descriptors that utilize statistical measures computed from regions in space, 
time, frequency and resolution. 

The Space and Frequency View (SFV) Description Scheme (DS) provides a way to 

specify regions in space, time, frequency and resolution in terms of space and frequency views. 

This allows a more rigorous definition and common understanding of terms such as 
YO999-043 3 



"half-resolution," "upper-right quadrant," or "high-pass band," such as when referring to views of 
an image. The SFV-DS also provides a way to index these views. This is important in 
applications of progressive retrieval in which a client application can send requests for views of 
images stored at a server in terms of an SFGraph index value. The SFV-DS handles the details 
concerning the access and relationship of views with different resolutions, spatial locations and 
sizes. 

Brief Description of the Drawings 

The invention will now be described in detail with specific reference to the appended drawings 
wherein: 

Figure 1 . Shows how the SFV-DS provides an abstraction layer between the multimedia 

applications and ether MPEG-7 ucSCilpliuu schemes (DS) and the storage and compression 
formats for the data. 

Figure 2. Shows the dependency among the SFV-DS object types. A tiling has a set of views, a 
complete tiling is a tiling that partitions the space and frequency multidimensional space, and 
a hierarchical decomposition has a number of tilings and/or complete tilings. 

Figure 3. Shows examples of space and frequency views, tilings and hierarchical 

decompositions: (a) 2-D lattice image data, (b) low-resolution image, (c) spatial segment of 
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image, (d) 4-subband frequency decomposition, (e) 4-segment spatial decomposition, (f) 
multi-resolution pyramid, (g) spatial quad-tree decomposition, (h) space and frequency graph. 

Figure 4. Shows the relationship between the space and frequency descriptors (D) in the 
SFV-DS. 

Figure 5. Shows a class diagram for AT-dimensional Space and Frequency View descriptor 
(SFV-D). 

Figure 6. Shows class diagrams for ^-dimensional Space and Frequency Tiling descriptor 
(SFT-D) and Space and Frequency Partitioning descriptor (SFP-D). 

Figure 7. Shows the relationship between the space and frequency hierarchical decomposition 
(SFHD) descriptors (D) in the SFV-DS. 

Figure 8. Shows the class diagrams for Af-dimensional Space and Frequency Hierarchical 
Decomposition (SFHD-D) and Space and Frequency Pyramid descriptor (SFPyr-D). 

Figure 9. Shows the class diagrams for ^dimensional Space and Frequency Tree descriptor 
(SFTree-D) and Space and Frequency Graph descriptor (SFGraph-D). 

Figure 10. Shows the Space and Frequency Graph (SFGraph), which provides a mechanism for 
indexing space and frequency views that are dyadic in location and size. Each view has an 
index value according to its position in the SFGraph. 

Figure 1 1 . Shows the Video View Element Graph, which indexes view with different resolutions 
in space and time. 
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Figure 12. Shows how the SFV-DS provides an interface between the SF-Views and the stored, 
compressed data stream. 
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Detailed Description of a Preferred Embodiment of the Invention 

The SFV-DS provides a standard way to describe locations, sizes, views, tilings and 
hierarchical decompositions in space, time, resolution and frequency. The SFV-DS also 
provides an abstraction layer between the multimedia applications that need multi-resolution 
sub-region access to image, video and audio data and the storage and compression formats for the 
data. This eliminates the need for the multimedia applications and the other specific MPEG-7 
description schemes (DSs) to be concerned with the underlying data representations and storage 
formats. This provides a higher-level interface in which the applications use the SFV-DS as a 
standard way of referring to space and frequency views. The SFV-DS has the responsibility of 
correlating those views with the specific storage and compression schemes. 

Figure 1 illustrates the abstraction enabled by the SFV-DS. SFV-DS acts as an interface 
(1 10) between different description schemes (DS) (105) and descriptors (D) within multimedia 
applications and as an interface (125) between the applications (120) and the data storage 
formats (130). Many of the multimedia applications use DSs and Ds that refer to locations and 
regions of the data in space, time, frequency and resolution. An Image-DS objects that point to 
sets of spatial regions of an image. A Video-DS proposal defines events that correspond to sets 
of segmented temporal units of video. Other descriptors (D), potentially describe features of 
regions of images, or scenes of video, or segments of audio. The SFV-DS (110) provides a 
standard way of defining locations, regions and sizes in space, time, frequency and resolution that 
enables the DSs (1 05) and Ds to be able to correlate with each other. 
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Furthermore, since the multimedia data can be potentially be stored in a large variety of storage 
and compression formats, the applications need to be insulated from the details of the storage and 
compression (115). For example, image data can be stored in raw pixel format, JPEG, Flashpix, 
Tiff, Gif, PNG, and many formats (115). Video and audio similarly can be stored in many 
different formats, such as MPEG-1 , MPEG-2, and so forth. The SFV-DS builds a view interface 
on top of the storage formats that translates the references to multi-resolution sub-region views in 
space and time into the appropriate references to and operations on the data streams of the 
storage and compression formats. 

Definitions 

In order to provide a common interface for specifying location, size, and regions in space, time, 
frequency and resolution SFV-DS defines several elements: SF-views (205), SF-tilings (210), 
SF-partitionings (215), and SF-hierarchical decompositions (220). The relationship between 
these elements is illustrated in Figure 2. The elements are Hefined in SF V -DS as follows: 

• SF-View: An SF-view (205) is a rectangular region of arbitrary size and location in 
multi-dimensional space and frequency. 

Two examples of views are shown in Figure 3. Figure 3(b) shows a low-resolution SF-view 
(310) of an image (305), which corresponds to a low-frequency region. Figure 3(c) shows a 
spatial SF-view (315), which corresponds to the upper-right quadrant of an image (305). We 
define an SF-tiling and an SF-partitioning as follows: 

• SF-Tiling: An SF-tiling (2 1 0) is a non-redundant set of views, where non-redundancy is 
measured in terms of overlap in space and frequency. 
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• SF-Partitioning: An SY -partitioning (215) is an SF-tiling that completely fills the 
multi-dimensional space in space and frequency. 

Together, the completeness and non-redundancy constraints imply that title set of SF-views 
provides a non-redundant and complete covering of the multi-dimensional space in space and 
frequency. Two examples of SF-partitionings are shown in Figure 3. Figure 3(c) shows a 
4-subband decomposition (320), which forms a uniform tiling in frequency. Figure 3(d) shows a 
4-segment spatial decomposition (325), which forms a uniform tiling in space. We define a 
SF-hierarchical decomposition as follows: 

• SF-Hierarchical decomposition: An SF-hierarchical decomposition (220) is a set of 
SF-tilings (210) and/or SF-partitionings (215) organized into a hierarchy, in which there 
is a processing dependency among the SF-views (205) at different levels in the hierarchy. 

Figure 3 shows three example hierarchical decompositions. Figure 3(f) shows a multi-resolution 
pyramid (330), which contains several versions of the image at different resolutions. This type of 
pyramid is used in uic JFEG progressive storage format for storing images at muitipie 
resolutions. Each level of the pyramid contains a tiling with a single view. Only the bottom 
tiling, full-resolution, is complete. Figure 3(g) shows a spatial quad-tree decomposition (335) in 
which the image is iteratively decomposed in space. Figure 3(h) shows a space and frequency 
graph (340), in which the image is iteratively decomposed in space and frequency. 

Definition of SFV-DS 

The SFV-DS provides a way to describe the SF-views, SF-tilings, SF-partitionings and 
SF-hierarchical decompositions in space and frequency as defined above. The SF-views are 
described by the SF-view descriptor (SFV-D) (405). The SF-tilings in space and frequency, 
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which can be represented by sets of SF-views } are described by the SF-tiling descriptor (SFT-D) 
(410). The SF-partitionings, which are specializations of SF-tilings, are described by the 
SF-partitioning descriptor (SFP-D) (415). The SF-hierarchical decompositions are described by 
the SF-hierarchical Decomposition descriptor (SFH-D) (420). Their relationship is shown in the 
class model depicted in Figure 4. 

Space and Frequency View (SFV) Descriptor (D) 

The SF-view descriptor (SFV-D) defines the object class SFView (505), which gives the location 
and size of a multi-dimensional rectangular region in space and frequency. The SF-views are 
meant to be general in the sense that they can have zero size, or full size in space or frequency. 
This allows them to be used to describe locations, durations, sizes and regions in space and 
frequency. Each SF-view is defined in terms of two N-dimensional rectangles, one in 
JV-dimensional space, and one in AT-dimensional frequency, where N is the dimensionality of the 
original data, where its dimensionality can be derived from the specific media type, for example, 
for audio, N= 1, for images, JV= 2, and for video, N = 3. 

The Rectangle class (510) defines the i^-dimensional rectangles with arbitrary location and size, 
where the these values are normalized with respect to the size of the space to fall in the range (0, 
1). The size value indicates the relative size of the rectangle with respect to the size of the space. 
The location value indicates the relative location of the rectangle with respect to the origin. The 
rectangle class also has a function intersect 0, which measures whether the rectangle intersects 
with another supplied rectangle. 
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The SFView class (505) has two rectangles: one in space (SRECT) and one in frequency 
(FRECT). The SFView class also has a function intersect 0, which measures the intersection of 
the view with another supplied view, where intersection is measured by the intersection of both 
the SRECT and FRECT rectangles of the SFView with those in the supplied SFView. 

Space and Frequency Tiling (SFT) Descriptor (D) 

The SF-tiling descriptor (SFT-D) (605) defines a set of SFViews. The number of SFViews in a 
SFTiling object is given by CNT. The set of SF-views can be complete or incomplete, but is 
necessarily non-redundant. The SFTiling class (605) has member functions that tests the 
SFView set for completeness and non-redundancy. The set of SF- views is complete if and only 
if the SF- views form a complete covering in ^-dimensional space and N -dimensional frequency. 
The set of SF-views is non-redundant if and only if the views do not intersect with each other. 

Space and Frequency Partitioning (SFP) Descriptor (D) 

The SF-partitioning descriptor (SFP-D) (610) specializes the SFTiling class (605). Since the 
SF-views form a complete cover in space and time, the SF-partitioning (610) can be defined from 
a single SF-view kernel. There are a number of ways in which the SF-view kernel can be 
resized, re-positioned and replicated in order to cover the multi-dimensional space. The TYPE 
data member gives two replication types: uniform or octave. This field can be extended to 
consider additional replication types. In TYPE=uniform, the kernel is replicated along each 
dimension in space and time until the multi-dimensional space is completely covered. In 
TYPE=octave, the tiling has logarithmically decreasing sized SF-views with increasing FRECT 
location. 
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Space and Frequency Hierarchical Decomposition (SFHD) Descriptor (D) 

The SF-hierarchical decomposition descriptor (SFHD-D) (720) forms the basis for pyramid 
(705), tree-structured (710) and graph-structured (715) decompositions in space and frequency. 

Space and Frequency Pyramid (SFPyr) Descriptor (D) 

The SFPyr class (810) holds an array of SFViews, which are ordered into a hierarchy related.to 
the processing required to generate the SFViews at subsequent levels in the hierarchy. The array 
index of each SFView gives its level in the SFPyr. 

Space and Frequency Tree (SFTree) Descriptor (D) 

Similarly, the SFTree class (905) holds an array of SFTilings, which are ordered into a hierarchy 
related to the processing required to generate the SFTiling at subsequent levels in the hierarchy. 
The array index of each SFTiling gives its level in the SFPyr. 

Space and Frequency Graph (SFGraph) Descriptor (D) 

Similarly, the SFGraph class (910) holds an array of SFTrees. The SFGraph forms a 
directed-acyclic graph (DAG) of SF-tilings in which the SF-views of each tiling are uniformly 
decomposed in space and frequency to generate the SF-views at the next level of the SFGraph. 
Since the SFGraph is grown uniformly in space and frequency, the SFGraph defines a space 
depth (SDEPTH) and frequency depth (FDEPTH). 

Space and Frequency Graph (SFGraph) Index 

The SFGraph also provides a way to index the SF-views in terms of their position in the 

SFGraph DAG data structure (see Figure 10). Each view, such as (1010) , that has position and 
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size that is a power of 2 can be represented by a single node in the SFGraph. For example, view 
(1010) can be assigned an index value that indicates its position in the SFGraph, in terms of 
depth in space and frequency and location in the SFTiling plane. This allows that view to be 
referred to on the basis of its index value. Views that do not have power of two in size and 
location can be represented by a set of nodes in the SFGraph. This way, an arbitrary rectagular 
view can be represented by a set of SFGraph index values. 

Video View Element Graph (VEG) Index 

In video applications, it is neccessary to de-couple spatial-resolution and temporal-resolution 
and organize the SF-views into a video view element graph hierarchical decomposition. Similar 
to the case of the space and frequency decomposition, the space and temporal resolution 
decomposition can be used to index space and temporal resolution views. For example, the view 
(1115) corresponds to a view of the video sequence with low-spatial and low-resolution. This 
view c?r> b ft ?.ssign^ ar » index value rm the basis of its location in the VEG. 

Space and Frequency View and Data Storage Interface 

As described in Figure 1, the SFV-DS can provide an abstraction layer between image, video and 

audio description schemes (DSs) and the stored, compressed data that allows the data to be 

referenced and accessed in terms of space and frequency views. An example table that contains 

the abstraction information is illustrated in Figure 12. Since the multimedia data can be 

potentially be stored in a large variety of storage and compression formats, the applications need 

to be insulated from the details of the storage and compression. The SFV-DS builds a SF-view 

interface on top of the storage formats that translates the references of multi-resolution 

sub-region views in space and time into the appropriate bit-ranges (1205) and operations on the 
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data streams. Each view (1210) can be assigned a list of bit-ranges (1205), which 
represent the data needed from the stored and compressed stream in order to extract 
the view. For example, view V0 (1215) can be accessed by accessing various 
segments of data from the bit-stream. The view to bit-stream translation table can 
potential store in addition a description of the processing function that is needed in 
order to generate the referenced view by processing the extracted bit-stream data. 

Extensions 

There are many other extensions that are possible that allow the Space and Frequency 
Description Scheme to include bit-plane, color channels and spectral information. 

Filters and Transforms 

Additional meta-data can be added to the Space and Frequency classes that describes the specific 
filters and transforms for generating the views, tilings, parititionings and hierarchical 
decompositions. 

Arbitrary shape multi-dimensional views 

Arbitrary shape multi-dimension views correspond to regions in space and frequency that are not 
necessarily rectangular. These shapes can be approximated by a set of SF Views at a particular 
resolution, or frequency depth in the SFGraph. Furthermore, by using SFViews from multiple 
resolutions, or frequency depths in the SFGraph, shapes can be defined at multiple levels of 
abstraction. 
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Descriptor attributes 

The SFViews can be used as an attribute in a descriptor or description scheme to specify the 
location and size of a region being referred to. For example, an image description scheme can 
specify a number of objects and their semantic attributes, such as names, colors, and shapes. An 
SFView description can be added to each object that gives its location in space, time, resolution 
and frequency. 

Space and Frequency Feature Descriptors 

The SFViews can be used in order to describe the derivation of feature descriptors that are 
computed from regions of the data in space, time, resolution and frequency. For example, an 
image texture descriptor may give a vector in which each dimension of the vector corresponds to 
a measure of energy of one of the spatial-frequency subbands of the image. In this case, each of 
the spatial-frequency subbands can be described as an SFView. This allows the descriptor 
derivation to be riven in terms nf a list of SFViews ?.nd ?. measurement function for ecich or ell of 
the SFViews, This provides a convenient and compact way to define new descriptors and 
include their derivation as part of the meta-data. 
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Figure 1 . Shows how the SFV-DS provides an abstraction layer between the multimedia 
applications and other MPEG-7 description schemes (DS) and the storage and compression 
formats for the data. 
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Figure 2. Shows the dependency among the SFV-DS object types. A tiling has a set of views, a 
complete tiling is a tiling that partitions the space and frequency multidimensional space, and a 
hierarchical decomposition has a number of tilings and/or complete tilings. 
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image SF-views SF-tilings SF-hierarchical decompositions 



Figure 3. Shows examples of space and frequency views, tilings and hierarchical 
decompositions: (a) 2-D lattice image data, (b) low-resolution image, (c) spatial segment of 
image, (d) 4-subband frequency decomposition, (e) 4-segment spatial decomposition, (f) 
multi-resolution pyramid, (g) spatial quad-tree decomposition, (h) space and frequency graph. 
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Figure 4. Shows the relationship between the space and frequency descriptors (D) in the 
SFV-DS. 
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SFView <N> 

DIM : {1, 2,3} = N 
SRECT : Rectangle <N> 
FRECT : Rectangle <N> 

intersect (SFView <N>) 

(505) ~ 



Rectangle <N> 

DIM :{ 1,2, 3} = N 
LOCATION : Float [N] = (0, 1) 
SIZE: Float [N] = (0, 1) 

intersect (Rectangle <N> ) 

(510) 



Figure 5. Shows a class diagram for N-dimensional Space and Frequency View descriptor 
(SFV-D). 
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SFPartitioning <N> : SFTiling <N> 



DIM: {1,2, 3} = N 

TYPE : {Uniform, Octave} 

KERNEL : SFView <N> 



(610) 



Figure 6. Shows class diagrams for JV-dimensional Space and Frequency Tiling descriptor 
(SFT-D) and Space and Frequency Partitioning descriptor (SFP-D). 
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Figure 7. Shows the relationship between the space and frequency hierarchical 
decomposition (SFHD) descriptors (D) in the SFV-DS. 
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SFPyr <N> : SFHierDecomp <N> 
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Figure 8. Shows the class diagrams for JV-dimensional Space and Frequency Hierarchical 
Decomposition (SFHD-D) and Space and Frequency Pyramid descriptor (SFPyr-D). 
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SFTree <N> : SFHierDecomp <N> 



SFTiling <H> : Tiling [LEVELS] 



IndexValue (SFView <N>) 
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Figure 9. Shows the class diagrams for AT-dimensional Space and Frequency Tree descriptor 
(SFTree-D) and Space and Frequency Graph descriptor (SFGraph-D). 
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Figure 1 0. Shows the Space and Frequency Graph (SFGraph), which provides a mechanism 
index value according to its position in the SFGraph. 
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Figure 1 1 . Shows the Video View Element Graph, which indexes view with different 
resolutions in space and time. 
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Deputy Commissioner 
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SUBJECT: Termination of transitional practice of supplying a copy of a 

provisional application relied upon to give prior art effect under 
35 U.S.C. § 102(e) to a reference applied in a rejection 

Recently, Internet access to provisional applications that are relied upon for their earlier 
filing dates in U.S. patent application publications or U.S. patents became available to the 
public via Public PAIR (Patent Application Information Retrieval). As a result, the 
transitional practice for supplying a copy of a provisional application relied upon to give 
prior art effect under 35 U.S.C. § 102(e) to a reference applied in a rejection has ended. 

The transitional practice began in December of 2003, and required examiners to: (1) list 
on the PTO-892 form the provisional application number that gives prior art effect to an 
applied reference; and (2) include form paragraph 7.82.02 in the written Office action. 
Now that the transitional practice has ended, these requirements are rescinded. 
Accordingly, form paragraph will soon be removed from OACS. 

Although the transitional practice has ended, whenever the examiner has a copy of a 
provisional application relied upon for prior art effect of an applied reference, the 
examiner may still include the copy with the Office action submitted for mailing (e.g., in 
the red Action folder). When a copy of a provisional application is included with the 
Office action submitted for mailing, the examiner should list the provisional application 
number in the bottom section of the PTO-892 form under the heading "Non-Patent 
Documents." 

If applicant requests a copy of a provisional application relied upon for prior art effect of 
a reference applied in an Office action, applicant should be referred to the Public PAIR 
website at http://TOrtal.uspto.gov/external/portal/Dair for viewing and/or printing the 
provisional application. If applicant cannot view or print the provisional application from 
the Public PAIR website, applicant can still use the Public PAIR website to order a copy 
of the provisional application. Whether the order for the provisional application copy is 
placed with the Office of Public Records directly from the Public PAIR website, or by 
mail (using Mail Stop Document Services), the order requires the fee under 37 CFR 
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